There appears to be a major memory leak in mod_ssl/OpenSSL

This is a multi-part message in MIME format.

------_=_NextPart_001_01C3E752.CA1743A6
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I have been tracking this down for a couple of weeks and thought it was
in the code my company is developing and it appears that is not the
case. In order to eliminate our code from the mix and isolate the
problem here is what I did:
This was done on Windows and Linux:=20
1. Download the latest Apache from www.apache.org.=20
2. Download the latest OpenSSL from www.openssl.org.=20
3. Build them both, with apache add the mod_ssl option and also for
Linux use the MPM worker module.=20
4. Install and modify the ssl.conf file ServerName value.=20
5. Run Apache (httpd)=20
6a. Run the Performance monitor on Windows and look at Private Bytes for
the second Apache process.=20
6b. On Linux run top -p pid(httpd1) -p pid(http2) ..... -p pid(httpN)
watching the size of the processes=20
7. Set you browser to not cache requests and check for a new page every
time.=20
8. Start fetching a page from https://localhost and keep refreshing the
page.=20
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all
verified that it leaks like a sieve.
We were all trying to figure out why no one else has complained about
such a huge leak so we ran another test. We tried using the prefork MPM
and it turns out that worked fine. Based on the results it appears the
OS is cleaning up memory for the prefork module and the threaded model
never gets its memory freed. I have used a debugger on Windows and set
break points on the CRYPTO_malloc and CRYPTO_free functions and have
seen gobs of memory CRYPTO_malloc(ed) and not one time have I seen
CRYPTO_free called. I was not sure if having the OS cleanup memory was
part of the design (if indeed that is what is happening) or if there is
potentially a problem in the OpenSSL memory management code.
With all this said, I am by no means an expert on this code and could
really use some help understanding what is going on here?
Any and all help is appreciated,=20
Ken=20


------_=_NextPart_001_01C3E752.CA1743A6
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable




charset=3Dus-ascii">
6.0.6487.1">

I have been tracking this down for a =
couple of weeks and thought it was in the code my company is developing =
and it appears that is not the case. In order to eliminate our code from =
the mix and isolate the problem here is what I did:

This was done on Windows and =
Linux:

1. Download the latest Apache =
from COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">www.apache.org FACE=3D"Arial">.

2. Download the latest OpenSSL =
from COLOR=3D"#0000FF" SIZE=3D2 =
FACE=3D"Arial">www.openssl.org FACE=3D"Arial">.

3. Build them both, with apache add =
the mod_ssl option and also for Linux use the MPM worker =
module.

4. Install and modify the ssl.conf =
file ServerName value.

5. Run Apache (httpd) FACE=3D"Times New Roman">

6a. Run the Performance monitor on =
Windows and look at Private Bytes for the second Apache =
process.

6b. On Linux run top -p pid(httpd1) =
-p pid(http2) ….. -p pid(httpN) watching the size of the =
processes

7. Set you browser to not cache =
requests and check for a new page every time. New Roman">

8. Start fetching a page from =
FACE=3D"Arial">https://localhost FACE=3D"Arial"> and keep refreshing the page. New Roman">


So far 3 other engineers have =
reproduced this test because they did not believe the problem could be =
in Apache mod_ssl/OpenSSL, they all verified that it leaks like a =
sieve.

We were all trying to figure out why no =
one else has complained about such a huge leak so we ran another test. =
We tried using the prefork MPM and it turns out that worked fine. Based =
on the results it appears the OS is cleaning up memory for the prefork =
module and the threaded model never gets its memory freed. I have used a =
debugger on Windows and set break points on the CRYPTO_malloc and =
CRYPTO_free functions and have seen gobs of memory CRYPTO_malloc(ed) and =
not one time have I seen CRYPTO_free called. I was not sure if having =
the OS cleanup memory was part of the design (if indeed that is what is =
happening) or if there is potentially a problem in the OpenSSL memory =
management code.

With all this said, I am by no means an =
expert on this code and could really use some help understanding what is =
going on here?

Any and all help is =
appreciated,

Ken Roman">

Re: There appears to be a major memory leak in mod_ssl/OpenSSL

On Fri, Jan 30, 2004 at 11:02:06AM -0600, Avery, Ken wrote:
> I have been tracking this down for a couple of weeks and thought it was
> in the code my company is developing and it appears that is not the
> case. In order to eliminate our code from the mix and isolate the
> problem here is what I did:
> This was done on Windows and Linux:
> 1. Download the latest Apache from www.apache.org.
> 2. Download the latest OpenSSL from www.openssl.org.
> 3. Build them both, with apache add the mod_ssl option and also for
> Linux use the MPM worker module.

Are you using 2.0.48? Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

Re: There appears to be a major memory leak in mod_ssl/OpenSSL

There appears to be a major memory leak in mod_ssl/OpenSSLHello,

I encountered many memory leak trouble with OpenSSL. I used quite old
versions (from 0.9.6c), so I don't know if this is relevant or not for
you... Anyway I figured out that nobody seemed to ever call the
CRYPTO_thread_cleanup(). I just added a call to that function, and the
memory heap became clean. I hope the solution will be that simple in your
case.

The OpenSSL code did not look robust to me on that matter... But I think
that OpenSSL people are trying to (or did) improve the cleanup code.

Adrien

----- Original Message -----
From: Avery, Ken
To: modssl-users@modssl.org
Sent: Friday, January 30, 2004 6:02 PM
Subject: There appears to be a major memory leak in mod_ssl/OpenSSL


I have been tracking this down for a couple of weeks and thought it was in
the code my company is developing and it appears that is not the case. In
order to eliminate our code from the mix and isolate the problem here is
what I did:
This was done on Windows and Linux:
1. Download the latest Apache from www.apache.org.
2. Download the latest OpenSSL from www.openssl.org.
3. Build them both, with apache add the mod_ssl option and also for Linux
use the MPM worker module.
4. Install and modify the ssl.conf file ServerName value.
5. Run Apache (httpd)
6a. Run the Performance monitor on Windows and look at Private Bytes for the
second Apache process.
6b. On Linux run top -p pid(httpd1) -p pid(http2) ... -p pid(httpN) watching
the size of the processes
7. Set you browser to not cache requests and check for a new page every
time.
8. Start fetching a page from https://localhost and keep refreshing the
page.
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all verified
that it leaks like a sieve.
We were all trying to figure out why no one else has complained about such a
huge leak so we ran another test. We tried using the prefork MPM and it
turns out that worked fine. Based on the results it appears the OS is
cleaning up memory for the prefork module and the threaded model never gets
its memory freed. I have used a debugger on Windows and set break points on
the CRYPTO_malloc and CRYPTO_free functions and have seen gobs of memory
CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free called. I was not
sure if having the OS cleanup memory was part of the design (if indeed that
is what is happening) or if there is potentially a problem in the OpenSSL
memory management code.
With all this said, I am by no means an expert on this code and could really
use some help understanding what is going on here?
Any and all help is appreciated,
Ken


---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system (http://www.grisoft.com).
Version: 6.0.574 / Virus Database: 364 - Release Date: 29/01/2004

____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: There appears to be a major memory leak in mod_ssl/OpenSSL

This is a multi-part message in MIME format.

------=_NextPart_000_0056_01C3E757.657A7100
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

There appears to be a major memory leak in mod_ssl/OpenSSLAre u using:

> On linux you really should be using a shared memory session cache - like
> SSLSessionCache shmcb:logs/ssl_gcache_data(512000)

> SSLSessionCacheTimeout 300


and not the dbm cache

I posted some email about this just before XMAS where I had found a "memory
leak" - and Mads Toftum suggested the use of shmcb. I then ran tests for
nearly a week - without a hint of a memory leak


John
-----Original Mess
age-----
From: owner-modssl-users@modssl.org
[mailto:owner-modssl-users@modssl.org]On Behalf Of Avery, Ken
Sent: 30 January 2004 17:02
To: modssl-users@modssl.org
Subject: There appears to be a major memory leak in mod_ssl/OpenSSL


I have been tracking this down for a couple of weeks and thought it was in
the code my company is developing and it appears that is not the case. In
order to eliminate our code from the mix and isolate the problem here is
what I did:

This was done on Windows and Linux:
1. Download the latest Apache from www.apache.org.
2. Download the latest OpenSSL from www.openssl.org.
3. Build them both, with apache add the mod_ssl option and also for Linux
use the MPM worker module.
4. Install and modify the ssl.conf file ServerName value.
5. Run Apache (httpd)
6a. Run the Performance monitor on Windows and look at Private Bytes for
the second Apache process.
6b. On Linux run top -p pid(httpd1) -p pid(http2) ... -p pid(httpN)
watching the size of the processes
7. Set you browser to not cache requests and check for a new page every
time.
8. Start fetching a page from https://localhost and keep refreshing the
page.
So far 3 other engineers have reproduced this test because they did not
believe the problem could be in Apache mod_ssl/OpenSSL, they all verified
that it leaks like a sieve.

We were all trying to figure out why no one else has complained about such
a huge leak so we ran another test. We tried using the prefork MPM and it
turns out that worked fine. Based on the results it appears the OS is
cleaning up memory for the prefork module and the threaded model never gets
its memory freed. I have used a debugger on Windows and set break points on
the CRYPTO_malloc and CRYPTO_free functions and have seen gobs of memory
CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free called. I was not
sure if having the OS cleanup memory was part of the design (if indeed that
is what is happening) or if there is potentially a problem in the OpenSSL
memory management code.

With all this said, I am by no means an expert on this code and could
really use some help understanding what is going on here?

Any and all help is appreciated,
Ken

------=_NextPart_000_0056_01C3E757.657A7100
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable



charset=3Dus-ascii">

style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px =
solid; MARGIN-RIGHT: 0px">

face=3DTahoma=20
size=3D2>-----Original Mess face=3DArial=20
color=3D#0000ff> 

face=3DTahoma=20
size=3D2> class=3D008213117-30012004> age-----
From:=20
owner-modssl-users@modssl.org =
[mailto:owner-modssl-users@modssl.org]On=20
Behalf Of Avery, Ken
Sent: 30 January 2004 =
17:02
To:=20
modssl-users@modssl.org
Subject: There appears to be a major =
memory=20
leak in mod_ssl/OpenSSL

I have been tracking this down for a =
couple of=20
weeks and thought it was in the code my company is developing and it =
appears=20
that is not the case. In order to eliminate our code from the mix and =
isolate=20
the problem here is what I did:

This was done on Windows and =
Linux: face=3D"Times New Roman">
1. =
Download the=20
latest Apache from href=3D"file://www.apache.org"> color=3D#0000ff=20
size=3D2>www.apache.org size=3D2>. face=3D"Times New Roman">
2. =
Download the=20
latest OpenSSL from href=3D"file://www.openssl.org"> color=3D#0000ff=20
size=3D2>www.openssl.org size=3D2>. face=3D"Times New Roman">
3. =
Build them both,=20
with apache add the mod_ssl option and also for Linux use the MPM =
worker=20
module.
face=3DArial=20
size=3D2>4. Install and modify the ssl.conf file ServerName =
value. face=3D"Times New Roman">
5. Run =
Apache=20
(httpd)
face=3DArial=20
size=3D2>6a. Run the Performance monitor on Windows and look at =
Private Bytes=20
for the second Apache process. face=3D"Times New Roman">
6b. On =
Linux run top=20
-p pid(httpd1) -p pid(http2) ….. -p pid(httpN) watching the size =
of the=20
processes
face=3DArial=20
size=3D2>7. Set you browser to not cache requests and check for a new =
page every=20
time.
face=3DArial size=3D2>8.=20
Start fetching a page from href=3D"https://localhost"> face=3DArial color=3D#0000ff =
size=3D2>https://localhost face=3DArial size=3D2> and keep refreshing the page. face=3D"Times New Roman">
So =
far 3 other=20
engineers have reproduced this test because they did not believe the =
problem=20
could be in Apache mod_ssl/OpenSSL, they all verified that it leaks =
like a=20
sieve.

We were all trying to figure out why no =
one else=20
has complained about such a huge leak so we ran another test. We tried =
using=20
the prefork MPM and it turns out that worked fine. Based on the =
results it=20
appears the OS is cleaning up memory for the prefork module and the =
threaded=20
model never gets its memory freed. I have used a debugger on Windows =
and set=20
break points on the CRYPTO_malloc and CRYPTO_free functions and have =
seen gobs=20
of memory CRYPTO_malloc(ed) and not one time have I seen CRYPTO_free =
called. I=20
was not sure if having the OS cleanup memory was part of the design =
(if indeed=20
that is what is happening) or if there is potentially a problem in the =
OpenSSL=20
memory management code.

With all this said, I am by no means an =
expert on=20
this code and could really use some help understanding what is going =
on=20
here?

Any and all help is =
appreciated, face=3D"Times New Roman">
size=3D2>Ken face=3D"Times New Roman">

RE: There appears to be a major memory leak in mod_ssl/OpenSSL

I have tried the shmcb and that does not help, I think the real issue
has to do with worker MPM verses prefork MPM. Note: Linux worker MPM and
Windows mpm_winnt MPM are threaded verses prefork MPM which has its own
memory space.

I will try the patches Joe recommended and see what happens.

Thanks Joe,
Ken

BTW - I am using 2.0.48, I just download the latest and see the problem.

-----Original Message-----
From: Joe Orton [mailto:jorton@redhat.com]=20
Sent: Friday, January 30, 2004 11:13 AM
To: Avery, Ken
Cc: modssl-users@modssl.org
Subject: Re: There appears to be a major memory leak in mod_ssl/OpenSSL


On Fri, Jan 30, 2004 at 11:02:06AM -0600, Avery, Ken wrote:
> I have been tracking this down for a couple of weeks and thought it=20
> was in the code my company is developing and it appears that is not=20
> the case. In order to eliminate our code from the mix and isolate the=20
> problem here is what I did: This was done on Windows and Linux:
> 1. Download the latest Apache from www.apache.org.=20
> 2. Download the latest OpenSSL from www.openssl.org.=20
> 3. Build them both, with apache add the mod_ssl option and also for
> Linux use the MPM worker module.=20

Are you using 2.0.48? Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org

RE: There appears to be a major memory leak in mod_ssl/OpenSSL

I just tried the 2 patches listed below and they did not make a
difference, using mpm_winnt and worker MPM.

-----Original Message-----
From: Joe Orton [mailto:jorton@redhat.com]=20
Sent: Friday, January 30, 2004 11:13 AM
To: Avery, Ken
Cc: modssl-users@modssl.org
Subject: Re: There appears to be a major memory leak in mod_ssl/OpenSSL


On Fri, Jan 30, 2004 at 11:02:06AM -0600, Avery, Ken wrote:
> I have been tracking this down for a couple of weeks and thought it=20
> was in the code my company is developing and it appears that is not=20
> the case. In order to eliminate our code from the mix and isolate the=20
> problem here is what I did: This was done on Windows and Linux:
> 1. Download the latest Apache from www.apache.org.=20
> 2. Download the latest OpenSSL from www.openssl.org.=20
> 3. Build them both, with apache add the mod_ssl option and also for
> Linux use the MPM worker module.=20

Are you using 2.0.48? Could be one of these two bugs:

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25667
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=3D25659

Try using the 'shmcb' session cache instead of dbm.

Regards,

joe
____________________________________________________________ __________
Apache Interface to OpenSSL (mod_ssl) www.modssl.org
User Support Mailing List modssl-users@modssl.org
Automated List Manager majordomo@modssl.org